Consolidated guidance for behavioral intervention pilot and feasibility studies.

BACKGROUND: In the behavioral sciences, conducting pilot and/or feasibility studies (PFS) is a key step that provides essential information used to inform the design, conduct, and implementation of a larger-scale trial. There are more than 160 published guidelines, reporting checklists, frameworks, and recommendations related to PFS. All of these publications offer some form of guidance on PFS, but many focus on one or a few topics. This makes it difficult for researchers wanting to gain a broader understanding of all the relevant and important aspects of PFS and requires them to seek out multiple sources of information, which increases the risk of missing key considerations to incorporate into their PFS. The purpose of this study was to develop a consolidated set of considerations for the design, conduct, implementation, and reporting of PFS for interventions conducted in the behavioral sciences. METHODS: To develop this consolidation, we undertook a review of the published guidance on PFS in combination with expert consensus (via a Delphi study) from the authors who wrote such guidance to inform the identified considerations. A total of 161 PFS-related guidelines, checklists, frameworks, and recommendations were identified via a review of recently published behavioral intervention PFS and backward/forward citation tracking of a well-known PFS literature (e.g., CONSORT Ext. for PFS). Authors of all 161 PFS publications were invited to complete a three-round Delphi survey, which was used to guide the creation of a consolidated list of considerations to guide the design, conduct, and reporting of PFS conducted by researchers in the behavioral sciences. RESULTS: A total of 496 authors were invited to take part in the three-round Delphi survey (round 1, N = 46; round 2, N = 24; round 3, N = 22). A set of twenty considerations, broadly categorized into six themes (intervention design, study design, conduct of trial, implementation of intervention, statistical analysis, and reporting) were generated from a review of the 161 PFS-related publications as well as a synthesis of feedback from the three-round Delphi process. These 20 considerations are presented alongside a supporting narrative for each consideration as well as a crosswalk of all 161 publications aligned with each consideration for further reading. CONCLUSION: We leveraged expert opinion from researchers who have published PFS-related guidelines, checklists, frameworks, and recommendations on a wide range of topics and distilled this knowledge into a valuable and universal resource for researchers conducting PFS. Researchers may use these considerations alongside the previously published literature to guide decisions about all aspects of PFS, with the hope of creating and disseminating interventions with broad public health impact.

Assessment of transparency indicators in space medicine.

Space medicine is a vital discipline with often time-intensive and costly projects and constrained opportunities for studying various elements such as space missions, astronauts, and simulated environments. Moreover, private interests gain increasing influence in this discipline. In scientific disciplines with these features, transparent and rigorous methods are essential. Here, we undertook an evaluation of transparency indicators in publications within the field of space medicine. A meta-epidemiological assessment of PubMed Central Open Access (PMC OA) eligible articles within the field of space medicine was performed for prevalence of code sharing, data sharing, pre-registration, conflicts of interest, and funding. Text mining was performed with the rtransparent text mining algorithms with manual validation of 200 random articles to obtain corrected estimates. Across 1215 included articles, 39 (3%) shared code, 258 (21%) shared data, 10 (1%) were registered, 110 (90%) contained a conflict-of-interest statement, and 1141 (93%) included a funding statement. After manual validation, the corrected estimates for code sharing, data sharing, and registration were 5%, 27%, and 1%, respectively. Data sharing was 32% when limited to original articles and highest in space/parabolic flights (46%). Overall, across space medicine we observed modest rates of data sharing, rare sharing of code and almost non-existent protocol registration. Enhancing transparency in space medicine research is imperative for safeguarding its scientific rigor and reproducibility.

The Subjective Interpretation of the Medical Evidence.

This Viewpoint discusses whether commonly occuring, sometimes divergent interpretations of new evidence's validity and usefulness should be explained when experts and guideline committees provide recommendations for screening or treatment.

Is society caught up in a Death Spiral? Modeling societal demise and its reversal.

Just like an army of ants caught in an ant mill, individuals, groups and even whole societies are sometimes caught up in a Death Spiral, a vicious cycle of self-reinforcing dysfunctional behavior characterized by continuous flawed decision making, myopic single-minded focus on one (set of) solution(s), denial, distrust, micromanagement, dogmatic thinking and learned helplessness. We propose the term Death Spiral Effect to describe this difficult-to-break downward spiral of societal decline. Specifically, in the current theory-building review we aim to: (a) more clearly define and describe the Death Spiral Effect; (b) model the downward spiral of societal decline as well as an upward spiral; (c) describe how and why individuals, groups and even society at large might be caught up in a Death Spiral; and (d) offer a positive way forward in terms of evidence-based solutions to escape the Death Spiral Effect. Management theory hints on the occurrence of this phenomenon and offers turn-around leadership as solution. On a societal level strengthening of democracy may be important. Prior research indicates that historically, two key factors trigger this type of societal decline: rising inequalities creating an upper layer of elites and a lower layer of masses; and dwindling (access to) resources. Historical key markers of societal decline are a steep increase in inequalities, government overreach, over-integration (interdependencies in networks) and a rapidly decreasing trust in institutions and resulting collapse of legitimacy. Important issues that we aim to shed light on are the behavioral underpinnings of decline, as well as the question if and how societal decline can be reversed. We explore the extension of these theories from the company/organization level to the society level, and make use of insights from both micro-, meso-, and macro-level theories (e.g., Complex Adaptive Systems and collapsology, the study of the risks of collapse of industrial civilization) to explain this process of societal demise. Our review furthermore draws on theories such as Social Safety Theory, Conservation of Resources Theory, and management theories that describe the decline and fall of groups, companies and societies, as well as offer ways to reverse this trend.

Best Practices for Data Management and Sharing in Experimental Biomedical Research.

Effective data management is crucial for scientific integrity and reproducibility, a cornerstone of scientific progress. Well-organized and well-documented data enable validation and building upon results. Data management encompasses activities including organization, documentation, storage, sharing, and preservation. Robust data management establishes credibility, fostering trust within the scientific community and benefiting researchers' careers. In experimental biomedicine, comprehensive data management is vital due to the typically intricate protocols, extensive metadata, and large datasets. Low-throughput experiments, in particular, require careful management to address variations and errors in protocols and raw data quality. Transparent and accountable research practices rely on accurate documentation of procedures, data collection, and analysis methods. Proper data management ensures long-term preservation and accessibility of valuable datasets. Well-managed data can be revisited, contributing to cumulative knowledge and potential new discoveries. Publicly funded research has an added responsibility for transparency, resource allocation, and avoiding redundancy. Meeting funding agency expectations increasingly requires rigorous methodologies, adherence to standards, comprehensive documentation, and widespread sharing of data, code, and other auxiliary resources. This review provides critical insights into raw and processed data, metadata, high-throughput versus low-throughput datasets, a common language for documentation, experimental and reporting guidelines, efficient data management systems, sharing practices, and relevant repositories. We systematically present available resources and optimal practices for wide use by experimental biomedical researchers.

Are the Risk of Generalizability Biases Generalizable? A Meta-Epidemiological Study.

Background: Preliminary studies (e.g., pilot/feasibility studies) can result in misleading evidence that an intervention is ready to be evaluated in a large-scale trial when it is not. Risk of Generalizability Biases (RGBs, a set of external validity biases) represent study features that influence estimates of effectiveness, often inflating estimates in preliminary studies which are not replicated in larger-scale trials. While RGBs have been empirically established in interventions targeting obesity, the extent to which RGBs generalize to other health areas is unknown. Understanding the relevance of RGBs across health behavior intervention research can inform organized efforts to reduce their prevalence. Purpose: The purpose of our study was to examine whether RGBs generalize outside of obesity-related interventions. Methods: A systematic review identified health behavior interventions across four behaviors unrelated to obesity that follow a similar intervention development framework of preliminary studies informing larger-scale trials (i.e., tobacco use disorder, alcohol use disorder, interpersonal violence, and behaviors related to increased sexually transmitted infections). To be included, published interventions had to be tested in a preliminary study followed by testing in a larger trial (the two studies thus comprising a study pair). We extracted health-related outcomes and coded the presence/absence of RGBs. We used meta-regression models to estimate the impact of RGBs on the change in standardized mean difference (ΔSMD) between the preliminary study and larger trial. Results: We identified sixty-nine study pairs, of which forty-seven were eligible for inclusion in the analysis (k = 156 effects), with RGBs identified for each behavior. For pairs where the RGB was present in the preliminary study but removed in the larger trial the treatment effect decreased by an average of ΔSMD=-0.38 (range - 0.69 to -0.21). This provides evidence of larger drop in effectiveness for studies containing RGBs relative to study pairs with no RGBs present (treatment effect decreased by an average of ΔSMD =-0.24, range - 0.19 to -0.27). Conclusion: RGBs may be associated with higher effect estimates across diverse areas of health intervention research. These findings suggest commonalities shared across health behavior intervention fields may facilitate introduction of RGBs within preliminary studies, rather than RGBs being isolated to a single health behavior field.

Impact of trial attrition rates on treatment effect estimates in chronic inflammatory diseases: A meta-epidemiological study.

The objective of this meta-epidemiological study was to explore the impact of attrition rates on treatment effect estimates in randomised trials of chronic inflammatory diseases (CID) treated with biological and targeted synthetic disease-modifying drugs. We sampled trials from Cochrane reviews. Attrition rates and primary endpoint results were retrieved from trial publications; Odds ratios (ORs) were calculated from the odds of withdrawing in the experimental intervention compared to the control comparison groups (i.e., differential attrition), as well as the odds of achieving a clinical response (i.e., the trial outcome). Trials were combined using random effects restricted maximum likelihood meta-regression models and associations between estimates of treatment effects and attrition rates were analysed. From 37 meta-analyses, 179 trials were included, and 163 were analysed (301 randomised comparisons; n = 62,220 patients). Overall, the odds of withdrawal were lower in the experimental compared to control groups (random effects summary OR = 0.45, 95% CI, 0.41-0.50). The corresponding overall treatment effects were large (random effects summary OR = 4.43, 95% CI 3.92-4.99) with considerable heterogeneity across interventions and clinical specialties (I2 = 85.7%). The ORs estimating treatment effect showed larger treatment benefits when the differential attrition was more prominent with more attrition in the control group (OR = 0.73, 95% CI 0.55-0.96). Higher attrition rates from the control arm are associated with larger estimated benefits of treatments with biological or targeted synthetic disease-modifying drugs in CID trials; differential attrition may affect estimates of treatment benefit in randomised trials.

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics.

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Smoking Cessation, or How to Avert Half a Billion Premature Deaths - Now.

An estimated 1.1 billion people currently smoke cigarettes,1 and 50 to 70% likely will die from tobacco-related causes.2 This translates to 550 to 770 million expected tobacco deaths among those who currently smoke. Many additional deaths will accrue in successive generations if the status quo continues. Of interest is the reversibility of the excess mortality risk of smoking. The meta-analysis by Cho et al.3 of four large national cohorts of nearly 1.5 million adults followed on average 14.8 years yielded 23.0 million person-years of observational data with over 120,000 deaths identified through linked death registries.

Estimating the extent of selective reporting: An application to economics.

Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as non-significant. We estimate the extent of selectively reported p-values to range between 57.7% and 71.9% of the significant p-values. The counterfactual p-value distribution also allows us to assess shifts of p-values along the entire distribution of published p-values, revealing that particularly very small p-values (p < 0.001) are unexpectedly abundant in the published literature. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies.

Large language models for science and medicine.

Large language models (LLMs) are a type of machine learning model that learn statistical patterns over text, such as predicting the next words in a sequence of text. Both general purpose and task-specific LLMs have demonstrated potential across diverse applications. Science and medicine have many data types that are highly suitable for LLMs, such as scientific texts (publications, patents and textbooks), electronic medical records, large databases of DNA and protein sequences and chemical compounds. Carefully validated systems that can understand and reason across all these modalities may maximize benefits. Despite the inevitable limitations and caveats of any new technology and some uncertainties specific to LLMs, LLMs have the potential to be transformative in science and medicine.

An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data.

BACKGROUND: Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies. METHODS: We compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve. RESULTS: Compared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately. CONCLUSIONS: Multiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only.

Methods proposed for monitoring the implementation of evidence-based research: a cross-sectional study.

OBJECTIVES: Evidence-based research (EBR) is the systematic and transparent use of prior research to inform a new study so that it answers questions that matter in a valid, efficient, and accessible manner. This study surveyed experts about existing (e.g., citation analysis) and new methods for monitoring EBR and collected ideas about implementing these methods. STUDY DESIGN AND SETTING: We conducted a cross-sectional study via an online survey between November 2022 and March 2023. Participants were experts from the fields of evidence synthesis and research methodology in health research. Open-ended questions were coded by recurring themes; descriptive statistics were used for quantitative questions. RESULTS: Twenty-eight expert participants suggested that citation analysis should be supplemented with content evaluation (not just what is cited but also in which context), content expert involvement, and assessment of the quality of cited systematic reviews. They also suggested that citation analysis could be facilitated with automation tools. They emphasized that EBR monitoring should be conducted by ethics committees and funding bodies before the research starts. Challenges identified for EBR implementation monitoring were resource constraints and clarity on responsibility for EBR monitoring. CONCLUSION: Ideas proposed in this study for monitoring the implementation of EBR can be used to refine methods and define responsibility but should be further explored in terms of feasibility and acceptability. Different methods may be needed to determine if the use of EBR is improving over time.

Personalized and longitudinal electronic informed consent in clinical trials: How to move the needle?

Changes in the clinical trials landscape have been driven by advancements in digital technology. The use of electronic informed consent to inform research participants and to obtain their consent electronically has the potential to improve participant-researcher interactions over time, facilitate clinical trial participation, and increase efficiency in clinical trial conduct. A personalized electronic informed consent platform that enables long-term interactions with the research team could function as a tool to empower participant engagement in clinical trials. However, significant challenges persist impeding successful and widespread implementation. This Perspective provides insights into the opportunities and challenges for the implementation of electronic informed consent in clinical trials. It sets out key recommendations to promote the implementation of this innovative approach to the informed consent process, including the creation of uniform electronic informed consent platforms at regional and national level.

Prognostic Biomarkers in Kidney Transplantation: A Systematic Review and Critical Appraisal.

SIGNIFICANCE STATEMENT: Why are there so few biomarkers accepted by health authorities and implemented in clinical practice, despite the high and growing number of biomaker studies in medical research ? In this meta-epidemiological study, including 804 studies that were critically appraised by expert reviewers, the authors have identified all prognostic kidney transplant biomarkers and showed overall suboptimal study designs, methods, results, interpretation, reproducible research standards, and transparency. The authors also demonstrated for the first time that the limited number of studies challenged the added value of their candidate biomarkers against standard-of-care routine patient monitoring parameters. Most biomarker studies tended to be single-center, retrospective studies with a small number of patients and clinical events. Less than 5% of the studies performed an external validation. The authors also showed the poor transparency reporting and identified a data beautification phenomenon. These findings suggest that there is much wasted research effort in transplant biomarker medical research and highlight the need to produce more rigorous studies so that more biomarkers may be validated and successfully implemented in clinical practice. BACKGROUND: Despite the increasing number of biomarker studies published in the transplant literature over the past 20 years, demonstrations of their clinical benefit and their implementation in routine clinical practice are lacking. We hypothesized that suboptimal design, data, methodology, and reporting might contribute to this phenomenon. METHODS: We formed a consortium of experts in systematic reviews, nephrologists, methodologists, and epidemiologists. A systematic literature search was performed in PubMed, Embase, Scopus, Web of Science, and Cochrane Library between January 1, 2005, and November 12, 2022 (PROSPERO ID: CRD42020154747). All English language, original studies investigating the association between a biomarker and kidney allograft outcome were included. The final set of publications was assessed by expert reviewers. After data collection, two independent reviewers randomly evaluated the inconsistencies for 30% of the references for each reviewer. If more than 5% of inconsistencies were observed for one given reviewer, a re-evaluation was conducted for all the references of the reviewer. The biomarkers were categorized according to their type and the biological milieu from which they were measured. The study characteristics related to the design, methods, results, and their interpretation were assessed, as well as reproducible research practices and transparency indicators. RESULTS: A total of 7372 publications were screened and 804 studies met the inclusion criteria. A total of 1143 biomarkers were assessed among the included studies from blood ( n =821, 71.8%), intragraft ( n =169, 14.8%), or urine ( n =81, 7.1%) compartments. The number of studies significantly increased, with a median, yearly number of 31.5 studies (interquartile range [IQR], 23.8-35.5) between 2005 and 2012 and 57.5 (IQR, 53.3-59.8) between 2013 and 2022 ( P < 0.001). A total of 655 studies (81.5%) were retrospective, while 595 (74.0%) used data from a single center. The median number of patients included was 232 (IQR, 96-629) with a median follow-up post-transplant of 4.8 years (IQR, 3.0-6.2). Only 4.7% of studies were externally validated. A total of 346 studies (43.0%) did not adjust their biomarker for key prognostic factors, while only 3.1% of studies adjusted the biomarker for standard-of-care patient monitoring factors. Data sharing, code sharing, and registration occurred in 8.8%, 1.1%, and 4.6% of studies, respectively. A total of 158 studies (20.0%) emphasized the clinical relevance of the biomarker, despite the reported nonsignificant association of the biomarker with the outcome measure. A total of 288 studies assessed rejection as an outcome. We showed that these rejection studies shared the same characteristics as other studies. CONCLUSIONS: Biomarker studies in kidney transplantation lack validation, rigorous design and methodology, accurate interpretation, and transparency. Higher standards are needed in biomarker research to prove the clinical utility and support clinical use.

Effectiveness of a fourth SARS-CoV-2 vaccine dose in previously infected individuals from Austria.

INTRODUCTION: Evidence is limited on the effectiveness of a fourth vaccine dose against coronavirus disease 2019 (COVID-19) in populations with prior severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections. We estimated the risk of COVID-19 deaths and SARS-CoV-2 infections according to vaccination status in previously infected individuals in Austria. METHODS: This is a nationwide retrospective observational study. We calculated age and gender adjusted Cox proportional hazard ratios (HRs) of COVID-19 deaths (primary outcome) and SARS-CoV-2 infections (secondary outcome) from 1 November to 31 December 2022, primarily comparing individuals with four versus three vaccine doses. Relative vaccine effectiveness (rVE) was calculated as (1-HR) X 100. RESULTS: Among 3,986,312 previously infected individuals, 281,291 (7,1%) had four and 1,545,242 (38.8%) had three vaccinations at baseline. We recorded 69 COVID-19 deaths and 89,056 SARS-CoV-2 infections. rVE for four versus three vaccine doses was -24% (95% CI: -120 to 30) against COVID-19 deaths, and 17% (95% CI: 14-19) against SARS-CoV-2 infections. This latter effect rapidly diminished over time and infection risk with four vaccinations was higher compared to less vaccinated individuals during extended follow-up until June 2023. Adjusted HR (95% CI) for all-cause mortality for four versus three vaccinations was 0.79 (0.74-0.85). DISCUSSION: In previously infected individuals, a fourth vaccination was not associated with COVID-19 death risk, but with transiently reduced risk of SARS-CoV-2 infections and reversal of this effect in longer follow-up. All-cause mortality data suggest healthy vaccinee bias.

Quantitative research assessment: using metrics against gamed metrics.

Quantitative bibliometric indicators are widely used and widely misused for research assessments. Some metrics have acquired major importance in shaping and rewarding the careers of millions of scientists. Given their perceived prestige, they may be widely gamed in the current "publish or perish" or "get cited or perish" environment. This review examines several gaming practices, including authorship-based, citation-based, editorial-based, and journal-based gaming as well as gaming with outright fabrication. Different patterns are discussed, including massive authorship of papers without meriting credit (gift authorship), team work with over-attribution of authorship to too many people (salami slicing of credit), massive self-citations, citation farms, H-index gaming, journalistic (editorial) nepotism, journal impact factor gaming, paper mills and spurious content papers, and spurious massive publications for studies with demanding designs. For all of those gaming practices, quantitative metrics and analyses may be able to help in their detection and in placing them into perspective. A portfolio of quantitative metrics may also include indicators of best research practices (e.g., data sharing, code sharing, protocol registration, and replications) and poor research practices (e.g., signs of image manipulation). Rigorous, reproducible, transparent quantitative metrics that also inform about gaming may strengthen the legacy and practices of quantitative appraisals of scientific work.

Principles for good scholarship in systematic reviews.

Many sources document problems that jeopardize the trustworthiness of systematic reviews. This is a major concern given their potential to influence patient care and impact people's lives. Responsibility for producing trustworthy conclusions on the evidence in systematic reviews is borne primarily by authors who need the necessary training and resources to correctly report on the current knowledge base. Peer reviewers and editors are also accountable; they must ensure that systematic reviews are accurate by demonstrating proper methods. To support all these stakeholders, we attempt to distill the sprawling guidance that is currently available in our recent co-publication about best tools and practices for systematic reviews. We specifically address how to meet methodological conduct standards applicable to key components of systematic reviews. In this complementary invited review, we place these standards in the context of good scholarship principles for systematic review development. Our intention is to reach a broad audience and potentially improve the trustworthiness of evidence syntheses published in the developmental medicine literature and beyond.